Entertainment system for performing human intelligence tasks

ABSTRACT

A game engine is configured to accept human intelligence tasks as in-game content and present the in-game content to the game player. A method performed by the game engine enables performance of human intelligence tasks, such as visual discrimination, in a video game context. The game engine may receive a definition of human intelligence tasks from one or more remote sources. The game engine may present the human intelligence tasks to multiple video game participants as in-game content. The game engine defines and enables game play rules for the in-game content. The game play rules set parameters for the multiple video game participants to perform the human intelligence tasks to achieve desired results. The game engine may award each of the multiple video game participants an improved game score upon successful performance of the human intelligence tasks in accordance with the game play rules. The game engine may measure success by consistency in responses between different participants or trials.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 15/443,984, filed on Feb. 27, 2017, now issued asU.S. Pat. No. 9,937,419, which claims priority to and is a continuationof U.S. patent application Ser. No. 13/532,675, filed on Jun. 25, 2012,now issued as U.S. Pat. No. 9,579,575, which claims priority to and is acontinuation of U.S. patent application Ser. No. 12/362,396, filed onJan. 29, 2009, now issued as U.S. Pat. No. 8,206,222, which claimspriority pursuant to 35 U.S.C. § 119(e) to U.S. provisional applicationSer. No. 61/024,347, filed Jan. 29, 2008, which applications arespecifically incorporated herein, in their entireties, by reference.

BACKGROUND 1. Field

The present disclosure relates to a gaming system that distributes tasksfor performance by human intelligence and collects task results.

2. Description of Related Art

Various processing tasks exist that are difficult to perform using anautomated algorithm, but that are relatively trivial for a humanoperator. For example, there is a substantial need for identification,characterization, or classification of features of photographs, soundsand other digital data used to produce visual or auditory Image output.This identification, characterization or classification either eludescomputerized systems or requires human confirmation of computerizedanalysis. At least one computerized system exists to distribute suchtasks to human operators in exchange for renumeration of some kind. Forexample, Amazon developed a system coined Mechanical Turk(http://www.mturk.com/mturklwelcome) that pays humans to perform HumanIntelligence Tasks. Mechanical Turk defines Human Intelligence Tasks as“simple tasks that people do better than computers.” As an example, aperson may be able to perform the task of identifying whether a specifictype of object (for example, a pizza parlor) appears in a photograph orvideo sequence easier and more efficiently than a computer.

This model, paying for people to perform tasks, fails where the cost ofthe labor to perform a task exceeds the value of the tasks. For example,a task may be to identify parking meters in a system similar to GoogleEarth™. The value to the company seeking the information may be only atenth of a penny per parking meter. A human operator may perform thetask a maximum of 250 times in an hour, on average. Performed by aperson, this task may not make financial sense, even when pricing laborin cheap offshore outsourced labor markets.

Complicating this problem, humans often make errors even in those tasksthat they are uniquely best suited to perform. Many errors arise throughcarelessness or just normal momentary lapses in concentration. Somepeople may intentionally enter incorrect data, either maliciously, or inorder to boost their pay rate by creating false results. Accordingly,there is a need for a system that effectively distributes tasks andcollects results for human intelligence tasks, for example visualidentification tasks, in a more cost-effective manner and in a mannerthat prevents or corrects erroneous entries.

Various distributed computing systems are known, in which a serverdistributes processing jobs to a plurality of clients for performance bythe client during processor idle time, and collects results. However,the processing tasks in those prior art systems are performed solely bythe client processor in cooperation with a server, and do not involve orrequire human intelligence input. In addition, various methods andsystems are known for distributing updated digital image or audio datato be output by a game engine during game play at one or more localclients, for advertising or game enhancement purposes. Some such priorsystems also collect use information regarding user interaction with theupdated data, e.g., number of views or clicks, and report the useinformation to a central server for analysis or control of distributedupdates. However, such systems do not attempt to solve any definedproblem through human interaction via game play. Problem solutionthrough game play requires unique methods and solutions that have notbeen contemplated in any prior art system. Indeed, it has not beencontemplated that game play can be used to solve problems requiringhuman intelligence input, especially problems involving theidentification, characterization or classification of visual images oraudible output based on qualities that humans are uniquely adapted torecognize, but that are difficult or impossible to recognize using anautomated algorithm.

SUMMARY

There are numerous video and other games where players navigate richvisual, audio or other environments. These games may involve singleplayer scenarios with many players using the same environment, but ininstances that do not include other players. Multiplayer and massivelymultiplayer virtual environments also exist. In general, systems fordistributing updated data, including digital data for visual images oraudible sounds output during game play, may be adapted for the solutionof problems benefiting from human intelligence input.

Before proceeding to solve such problems, they must be defined prior tosystem input, and broken down to a solution process. The solutionprocess presents digital data as visual or audible output for humaninteraction in the context of game play, and analyzes game play input toinfer the presence or absence (or probability of presence or absence) ofsome defined quality in discrete pieces of the digital image date. Thediscrete pieces of data may be perceivable in system output as stillimages, video clips, sound clips, or some combination of the foregoing.In general the digital image data may be derived from real-worldimagery, for example, photographs, video clips, sound recordings, x-rayimages, magnetic resonance images, ultrasound images, or any imagery ofpersons or objects or objects existing in the real world. Such imageryis generally more likely to include human-recognizable qualities thatcannot readily be discerned by automated analysis, as compared tocomputer-generated data that may be more readily characterized bymachines than by humans. The problem definition involves determining thedigital image data to be processed, the quality or qualities to bedetermined by human intelligence input during distributed game play, andthe desired outputs. A problem definition unit may comprise a serveroperating a user interface and application that presents possibleproblem parameters and receives input indicating a selection ofparameters defining the problem to be solved. A few examples, of suchproblem parameters are presented in the instant disclosure, which shouldnot be regarded as limiting to the scope of what is claimed.

Once the problem parameters have been defined, a solution generation andadministration system may define and operate a solution system designedto operate over a distributed hardware network. Generally the hardwarenetwork includes at least one server for distributing the digital imagedata to a plurality of client devices and client devices at which humanintelligence input is received in response to visible or audible outputat the client device, during game play. The components of the networkare in communication using any suitable network, including but notlimited to the Internet, a local area network, cellular telephonenetwork, satellite, cable, or optical fiber network, or some combinationof the foregoing. The solution system operating on this hardware networkmay include server and client application modules designed to receiveand distribute the digital image data, receive human intelligence inputduring game play, provide the input for automated analysis to inferproblem solution data, and report solution status and results. Solutionsystems may be generated manually by a solution engineer, may beautomatically generated to work with constrained problem parameterswithin a defined hardware and software system, or some combination ofmanual and automatic generation.

The solution system should be designed to address a unique aspect ofproblems depending on human intelligence input, and particularly onproblems depending on identification, characterization, orclassification of sensible output according to human-recognizable terms.Namely, the unique aspect that a correct solution depends on correcthuman intelligence input. At the same time, correctness or reliabilityof a solution cannot readily be assessed without confirming humanintelligence input. If it were otherwise, there would be no need forhuman intelligence input to solve the particular problem. In addition,problem solving by humans inherently involves a high error rate, eitherintentional or unintentional. Therefore, a robust solution system shouldinclude a confirmation procedure for the human intelligence input, suchas, for example, presenting identical digital image data at differenttimes on the same client, and/or at different clients, and assessingconsistency between the response inputs received. In general, a highdegree of consistency between responses may be used to indicate acorrect result, while a high degree of inconsistency correlated to anincorrect or indeterminate result, which may call for further solutionor a conclusion that a particular result is indeterminate or uncertain.For example, in some cases it may not be clear just what a particularimage represents. Furthermore, identifying a particular image as beingindeterminate may make submission of those indeterminate images to priorart systems (such as Mechanical Turk) cost-effective, as the value of acomplete, fully determinate set of images may be higher than the cost ofidentifying the several remaining indeterminate images afterapplications of the inventions disclosed herein.

Another unique aspect of the problems addressed by the presentdisclosure is that the human intelligence task to be performed on eachpiece of human task data may be of such low economic value that apayment-for-services model is not economically feasible. In thealternative, it may be simply desired to save cash requirements or othercosts associated with such a model. Furthermore, certain tasks areperformed by human brains differently when done in a competitiveenvironment, when done for pleasure, when done in a setting or mannerthat triggers the fight-or-flight response, or when done for reasonsother than remuneration. For example, people may be more likely to trusttheir instinctive guess as to the fastest path between two points whenin a competitive environment where time spent calculating the fasterpath would reduce the likelihood of victory. It is therefore desirableto solicit and receive the human intelligence input from multiplepersons, such as from a massive group of participants, without payingfor the input and while still providing the participants with anincentive to participate and to provide correct input. It is known thatlarge groups of participants will willingly participate in online gamingfor the entertainment value afforded by the game, the satisfaction ofbesting others or ones own past achievements by more adroitly performinggame requirements, or other intangible benefits. The game system maytherefore harness the motivations present in game play to motivateparticipation in evaluating the human task data, and reward inputindicating human intelligence input within the context of the game'sreward or scoring system. It should be understood that games or eithervirtual environments that simulate other people, or even single playergames or virtual environments, can also serve as a source of humanintelligence input.

Using at least one piece of data for which human identification or otherprocessing is required (“human task data”), a solution system may injectone or more pieces of human task data into a computer game in a mannerthat includes task data output in play according to the goals of thegame. Human task data may include digital image data or other datacapable of producing sensible output for which human identification,characterization, or classification is desired. The game may be designedsuch that user input interacting with image or auditory output appearingin the game provides information about a classification into which aparticular piece of human task data falls, or an identification orcharacterization that pertains to the particular piece of human taskdata.

For example, it may be desirable to identify the breed of dogs appearingin photographs found on the internet. A programmer may write softwarethat identifies potential dog images, which may be operated to collectsuch images from an available database. Alternatively, photographs foundon pages with text about dogs, or photographs bearing a file nameindicating a dog may be gathered. These images (or the portions thereofthat are most likely to be dogs) may be loaded into a walk-through“first person shooter” game. The walkthrough may be performeddynamically or not dynamically. Players may be told that shooting a dogof a specified breed will result in extra points, a higher score,in-game currency, extra ammunition or other in-game rewards within thegame context.

Rewards may be given immediately for shooting any human task data, orthe rewards may be delayed until after processing of player behaviordata takes place. The behavior of players to the injected to human taskdata may then be recorded, optionally transmitted to a centralized dataprocessing facility, and analyzed. If a set number of players “shoot”the dog in image 205 when instructed to shoot a pit bull or a fixedratio of players do so, the photograph is identified as a pit bull. Thisassumes also that certain players misidentify the photograph.Accordingly, it may be desirable to use a percentage, such as 80%, forexample. The photograph may then be rotated out and/or the players whohave “shot” the dog correctly may be given their reward. Thus, scoresand rewards may be awarded after the system has measured consistencybetween a number of user responses to infer a result that a particularimage is indeed a pit bull. This analysis may be performed during gameplay so that the points can be awarded during game play or no later thanat the completion of the game when final results are tallied. Otherrewards may be awarded prior to analysis. For example, the game mayimmediately award one point for “shooting” any image presented, andlater award five points if the image shot is the same as shot by amajority or super-majority of the participants. An additional incentivecan be provided to the first player to correctly identify an image, suchas by later awarding reputation points unrelated to the game (such asMicrosoft's Xbox Live Gamer Score points). Such an additional incentivecould be utilized to offset the disincentive that may be created when animage or task is correctly performed but the system lacks sufficientdata to verify the player input and award points based thereon. Anotherimplementation may permit users to undo an incorrect identification (forexample, as by clicking an “oops” button after shooting an image of asquirrel when they are tasked with shooting a dog), and may optionallyadjust the data utilized by the system and/or the user's score toreflect this correction.

A particularly vexing problem for mathematicians and planners is the“travelling salesman's problem”, one of a class of problems known as“NP-complete problems”. In one classic example of the problem, asalesman is required to visit N locations in the most time ordistance-efficient manner. However, finding the shortest routeconnecting N points, while solvable by testing every possiblecombination, is so computationally intensive that it is generallyconsidered to be unsolvable for large numbers of points using existingcomputing resources. This problem is particularly well suited to theinventions at issue. Human beings spend hundreds of millions of hoursannually navigating virtual 3-D environments within computer games. Byadapting the map for which the shortest route is sought to a computergame, players would be presented with a digital model of the locationsto be visited. The game is structured to require the players to travelbetween the points, and the relative efficiency of the various routingcombinations tried by the players is tracked by the software. Real worldfactors, such as traffic flow, stop signs, one way streets, speedlimits, and other things that impact transit speed can be accounted forby altering the routes within the game (as by lengthening game routesthat correspond to lower speed limit real-world streets) and/or byaltering the physics used by the game engine or altering the game map inways that would not be possible within the real world. One suchimplementation might be to stretch the area of the game mapcorresponding to the city center (where real world speed limits are low)by different amounts depending on the time of day, with or withoutexpanding surrounding areas of the map (as it is not necessary for gamemaps to follow laws of physics, allowing for example, a 5 square milearea to contain 10 square miles of territory). Points can be awarded toplayers for efficiently navigating the route. Alternatively, speed ofnavigation may simply be incorporated within the game as necessary oruseful to a good game score or outcome.

A more complete understanding of the entertainment system fordistributing and processing human intelligence tasks will be afforded tothose skilled in the art, as well as a realization of additionaladvantages and objects thereof, by a consideration of the followingdetailed description. Reference will be made to the appended sheets ofdrawings which will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart showing exemplary steps of a method for obtainingand processing human intelligence input that may be performed by anentertainment system for solving problems using human intelligenceinput.

FIG. 2 is a system diagram showing exemplary components of anentertainment system for solving problems using human intelligenceinput.

DETAILED DESCRIPTION

FIG. 1 is a flow diagram showing exemplary steps of a method 100 ofperformance of human intelligence tasks in a video game context, as maybe performed by an entertainment system for solving problems using humanintelligence input. An exemplary system is described later in thespecification. At 110, the system may receive human intelligence tasksfrom multiple remote sources. The human intelligence tasks may compriseidentifying photographs or parts of photographs, identifying sounds, oridentifying or otherwise indicating any type of audio, video, routing orother type of data. The multiple remote sources may be businesses,online resources, or any other source that requires human intelligenceto perform tasks.

At 120, the system may present the human intelligence tasks to multiplevideo game participants as in-game content. The in-game content may bequasi-content, meaning that the video game programmers did notspecifically include the content as part of the video game. Rather, thehuman intelligence tasks may be adapted to fit the video game usingreal-life images, sounds or other data, and may be updated at varioustimes after a game is coded and released. Various methods and systemsfor providing image, video and audio data after game release andincluding it in output during game play are known in the art, and anysuitable method or system may be used. However, such prior art systemsshould be adapted such that the video, image or audio data is presentedin a game context that solicits user input indicative of humanintelligence input regarding the video, image, or audio output from thedata.

At 130, the system may enable game play rules for the in-game content.The game play rules set the parameters for the multiple video gameparticipants to perform the human intelligence tasks to achieve desiredresults within the context of game play. The game play rules may, forexample, instruct the multiple video game participants in a“first-person shooter” video game to shoot only dogs of a specificbreed. Once any given video game participant has achieved the desiredresults in accordance with the game play rules, the video gameparticipant may receive a reward. At 140, the system may notify themultiple video game participants of the game play rules. At this point,the multiple video game participants may choose to participate in thein-game content for in-game or real-world rewards.

At 150, the system may reward the multiple video game participants uponsuccessful performance of the human intelligence tasks in accordancewith the game play rules. Successful performance may be measured by ameasured degree of consistency of any action or set of actions withinthe video game by a given video game participant, as compared to actionsmy other participants responsive to the same data, or to prior actionsby the same participant. The rewards may include in-game benefits,scores, prizes or a form of currency used within the video game, or mayinvolve real-world benefits, prizes or currency.

FIG. 2 is a block diagram illustrating an exemplary system 200 inaccordance with the present disclosure. The system 200 may comprise anetwork host computer 204, a plurality of clients 206, a database server208 and a database 210 all in communication via a Wide Area Network(WAN) 202. The WAN 202 may enable communication between the network hostcomputer 204, the plurality of clients 206, the database server 208 andthe database 210, and any suitable communication network, or combinationof networks, may be used. The network host computer 204 may comprise acontent management application 212, which may be encoded oncomputer-readable media and configured for performing various actions asillustrated in the flowchart of FIG. 1. In the alternative, or inaddition, each of the plurality of clients 206 may comprise a contentmanagement program 214, which may also be encoded on computer-readablemedia and configured for performing various actions illustrated in theflowchart of FIG. 1. Some of the actions illustrated in the flowchart ofFIG. 1 may be performed by the content management application 212, whileothers may be performed by the content management program 214. Thedatabase server 208 and attached database 210 may be coupled to thenetwork host computer 204 to store the database used in the methodillustrated in the flowchart of FIG. 1. Alternatively, the databaseserver 208 and/or database 210 may be connected to the WAN 202 and maybe operable to be accessed by the network host computer 204 via the WAN202.

The plurality of clients 206 may further comprise an internal hard diskor other storage device 216 for storing the game engine 214, a processor218 for executing the game engine 214 and/or performing other backgroundtasks and an internal bus 220 for internally connecting the storagedevice 216 and the processor 218. The storage device 216 may also beconfigured to store the database used method 100. The outputs of themethod illustrated by the flowchart of FIG. 1, the notification ofviolation of the guest requirements and termination of guest access, maybe displayed on the clients 206 via a display 222.

In accordance with the foregoing, system 200 comprises a server 204configured for distributing digital image data to a game client 206 viaa computer network 202. The digital image data may represent visible oraudible images of physical objects to be output during game play at theclient.

The game client 206 is in communication with the server, and comprises amemory 216 holding the game engine 214. The game engine may beconfigured to operate on the game client 216 output the digital imagedata as part of game output in response to receiving the digital imagedata from the server. The game client may receive and store the digitalimage data during game play, or prior to game play. The game engine mayfurther be configured to contemporaneously output a game environment incoordination with and exclusive of the digital image data. That is, thegame environment includes output data that is distinct from the digitalimage data, for example, background images, icons, sprites, avatars,menu screens, score and status data, and other data as known in thecomputer gaming arts. Also, the game environment operates incoordination with the digital image data, such that visible or audibleoutput generated from the digital image data is output by the gameclient as an integrated part of game play.

The game engine 214 is further configured to modify a game reward statusindicator, for example, a game score, responsive to user input receivedby the game client from a user interface device 224 during game play.Examples of user interface devices include keyboards, touch screens,pointers, game controllers, microphones and pointing devices. The gameengine is configured such that the modification to the game rewardstatus correlates to a degree of consistency in human discriminationbetween images or audio clips included in the digital image data, as maybe inferred from the user input. For example, more score points may beawarded for input consistent with that received from other clients forthe same data, then for inconsistent input. Conversely, points may bededucted for inconsistent input. The game engine may be furtherconfigured to output a record indentifying a sequence of the imagesoutput during game play correlated to the user input, the recordsufficient for inferring a predetermined attribute of ones of theimages.

The game client is may be further configured to transmit the record forinferring a predetermined attribute to the server. The server may beconfigured to receive individual game records from the different gameclients. The server may compare such records during game play to assessconsistency between responsive inputs received at different gameclients. The server may then report on the measured consistency to thegame clients, each of which may use the consistency data reported by theserver to generate a game score or other game reward status indicator,prior to completion of the game. In the alternative, the server maycompute a score or other game reward status indicator and report thecomputer status indicators or scores to the participating game clients.

In addition, the server may be configured to process the record using aninference algorithm to infer a probability that a predeterminedattribute applies to the ones of the images. The predetermined attributemay be an identity of a person or object appearing in the particularimages of the digital image data. For example, the attribute maycomprise the name of a person, species of animal, or object name (e.g.,“fire hydrant”). The predetermined to attribute may be a characteristicof a person or object appearing in the particular images of the digitalimage data. For example, the attribute may comprise an emotional stateindicated by a person's face or body language, for example, happy orsad, or a location where a photograph was taken, e.g., “London” or“Paris”. The predetermined attribute may be a classification of a personor object appearing in the particular images of the digital image data.For example, the attribute may comprise a label for various human orother classifications, for example, elderly, child, young, sexy, ugly,beautiful, fat, thin, and so forth. The server may assess a probabilitythat a particular attribute applies to a particular image, video clip,or audio clip based on a number and/or percentage of consistentresponses. For example, if 90% of responses indicate that a particularimage is of a “beautiful woman,” the server may infer that there is ahigh probability that the image indeed shows a beautiful woman.Conversely, for example, if only 30% of the responses agree that theimage is of a “beautiful woman,” the inferred probability assigned bythe server may be quite low.

In the alternative, or in addition, the game client may be configured toprocess the record using an inference algorithm to infer a probabilitythat a predetermined attribute applies to the ones of the images. Thismay be appropriate, for example, when a particular client has multipleusers or when it is difficult to maintain reliable communications withthe host server 204.

System 200 may further include an image processor 226 coupled to theserver. The image processor 226 may be configured for generating digitalimage data from photographic images or other real-world recorded data228. The image processor may obtain the data 228 from any availabledatabase, including database 208. Input image data may also be obtainedby searching any available records, for example using an Internet searchengine to identify candidate images of a particular person, object,place, or the like. The image processor may convert data 228 to a formatsuitable for output by game engine 214 during game play.

As part of developing a solution system to a particular problem, theserver 204 may be configured to define one or more attributes that areto be determined for images in the digital image data. These attributesmay be of a type as previously discussed, and are to be determined inresponse to user input received by the game clients 206 during via userinput devices 224. Another part of solution system development includesgenerating task definition information. For example, a task definitionmay include an identification of eligible input images and an attributelabel to the attribute to be determined from human intelligence input,for example, “toddler.” The task definition may include one of moreattributes to be determined. In addition, the task definition mayspecify other task criteria, for example a minimum number ofparticipants and views per image, a task completion date, eligible gameversion or application for gathering human intelligence input, and soforth. The server 204 may be configured to distribute the taskdefinition information to the game client 206. The game engine may beconfigured to output a description of the one or more attributesprovided by the task definition during game play. For example, the gameengine may output a message instructing players to take specific actionswith respect to images having specific attributes, to earn bonus pointsor other rewards. In general, the game engine is responsive to the taskdefinition to receive and process the digital image data as output forgame play.

The server 204 may be configured to divide the digital image data intodistinct sets, and to distribute different ones of the distinct sets todifferent client machines. In other words, each client may receive adifferent part of the digital image data. These different parts may beoverlapping. In the alternative, each client may receive the samedigital image data. The server may distribute the digital image data ina single batch to the game client prior to commencement of game play. Inthe alternative, or in addition, the server may distribute the digitalimage data to the game clients meted out into a sequence of batchesduring game play. The digital image data may be pushed to the clients bythe server, or pulled by the clients from the server.

The server 204 may distribute the digital image data via operation of amultiplayer game host operating on the server. The multiplayer host mayoperate in communication with the game engine on the game client toprovide a multiuser online game in which multiple participants interact.In such embodiments, the user input indicating human intelligenceresponse to particular task data may be provided directly to the server204 by the participating game clients. The server may then perform allassessment tasks directed towards problem solution or scoring. Ingeneral, the computational and data processing operations necessary foroperation of the solution system may be distributed between the clientand server in any appropriate fashion.

Having thus described a preferred embodiment of and entertainment systemfor distributing and processing human intelligence tasks, and method ofoperating the system, it should be apparent to those skilled in the artthat certain advantages of the within system have been achieved. Itshould also be appreciated that various modifications, adaptations, andalternative embodiments thereof may be made without departing from thescope and spirit of the present technology. The following claims definethe scope of what is claimed.

What is claimed is:
 1. A system comprising: a server; one or moreprocessors; a memory operably coupled to the server, the memory holdinginstructions that when executed by the one or more processors, cause thesystem to: receive, on a first computer, a request for confirmation thata user of a second computer is a human; automatically generate acomputer game consisting of a plurality of game components, where thegame components are one or more of still images, video clips and soundclips; request that a user of the second computer identifyhuman-recognizable qualities that cannot readily be discerned byautomated analysis; receiving, at the first computer, the identifiedhuman-recognizable qualities; comparing the identifiedhuman-recognizable qualities with a database of human-identifiedqualities in the game components; and verifying whether the identifiedhuman-recognizable qualities received at the first computer match thedatabase of human-identified qualities in the game components.
 2. Thesystem of claim 1, where the database of human-identified qualities inthe game components is generated by gathering data generated by humanusers of the system.
 3. The system of claim 1, where the game componentsare still images.
 4. The system of claim 1, where the game componentsare video clips.
 5. The system of claim 1, where the game components aresound clips.
 6. The system of claim 1, where the computer game consistsof identifying the contents of still images.
 7. A system comprising: aserver; one or more processors; a memory operably coupled to the server,the memory holding instructions that when executed by the one or moreprocessors, cause the system to: generate a plurality of gamecomponents, where the game components are one or more of still images,video clips and sound clips; incorporate the game components into acomputer game where the game logic requires identification of at leastone characteristic of at least one of the game components; receiving theidentification of the at least one characteristic; recording theidentification of the at least one characteristic, and one of the atleast one of the game components or an identifier for the at least oneof the game components; and storing the recorded information in adatabase.
 8. The system of claim 7, where the database is accessed todetermine whether input received from a computer identifying the atleast one game component has correctly identified the at least one gamecomponent.
 9. The system of claim 7, where the game components are stillimages.
 10. The system of claim 7, where the game components are videoclips.
 11. The system of claim 7, where the game components are soundclips.
 12. The system of claim 7, where the computer game consists ofidentifying the contents of still images.
 13. The system of claim 12,where identifications of the contents of still images are compared toeach other to determine which identification is most likely correct. 14.The system of claim 7, where the identification is done multiple times.15. A system comprising: a server; one or more processors; a memoryoperably coupled to the server, the memory holding instructions thatwhen executed by the one or more processors, cause the system to:automatically modify a computer game to include a plurality of gamecomponents, where the game components are one or more of still images,video clips and sound clips; and rewarding the user with an in-gamereward for identifying human-recognizable qualities of the gamecomponents that cannot readily be discerned by automated analysis. 16.The system of claim 15, where the game components are still images. 17.The system of claim 15, where the game components are video clips. 18.The system of claim 15, where the game components are sound clips. 19.The system of claim 15, where the computer game consists of identifyingthe contents of still images.
 20. The system of claim 15, where theidentifying is done multiple times.